L1 taxonomy: classify cancellations and cascade-only TCPStore logs - #439

Thirteen descriptions in l1/categories.json, plus one name. No decision value
changes and no code changes; every category keeps the action it had.

Each change traces to a production misclassification, read back to the wording
that caused it.

Cancelled jobs were recommended STOP. 4285548 and 4285549 were cancelled by
hand; their logs hold 751 SIGTERMs, 847 "Exception ignored" and healthy
iterations to the last line before the signal, and the grounded primary failure
is Python's own weakref finalizer losing a module global during interpreter
teardown - which Python labels Exception ignored precisely because it did not
affect execution. Cat 28 names "TypeError/AttributeError on None" and matched
that text; cat 1, the right answer, read in full "no application fault", which
is false on its face since a kill leaves plenty of application-shaped noise.
Cat 1 now carries both surfaces a cancellation leaves - the bare slurmstepd
marker, which is the entire content of some logs, and progress-then-teardown -
and cat 28 excludes tracebacks demonstrably raised during shutdown.

Cat 28's exclusion is limited to shutdown frames rather than lateness. A
deterministic fault can surface late: 4411569 cycle 3 failed after an automatic
scale-up set data_parallel_size to 8192 and Megatron asserted 12288 was not
divisible by 1 * 8192, on 8321 ranks with no teardown frames at all. That
recurs under the resulting configuration, so it stops.

Cascade-only TCPStore logs returned category 0 at confidence 0, telling a
reader nothing about a job that had checkpointed cleanly and then lost its
store. Cat 12 was the longest description of the 38 at 680 characters against a
median of 142, and the only one written in prohibitive voice. Completed
iterations settle these: a rendezvous failure cannot happen at iteration 48310
because rendezvous already succeeded. Cat 12 is gated on no progress; cat 3
owns the mid-training case.

Cat 21 read "Assertion after checkpoint resume (progress-log bookkeeping)" in
116 characters, which made it a superset of cat 28 for any post-resume
assertion - 4411569 landed there in 4 of 5 runs. It now names the bookkeeping
meant and excludes configuration-value assertions.

Cats 34 and 35 described the seed incident that produced them, pinning a rank
and a checkpoint number. Nine descriptions asserted the action as well as the
evidence; the decision field already carries that, and the description was a
second, unenforced copy. Claims about the nature of a failure are kept, since
that is what a classifier should weigh.

Validated on 26 live logs re-analyzed on an isolated instance, 13 labeled
dataset cases covering every edited category with positive and negative
controls, and individual inspection of 20 incidents. Five cancellations move
STOP to RESTART; 4188567 correctly stays STOP because its kill followed a real
MissingEpochIndex dataset fault; 4411569 moves RESTART to STOP; the labeled
TCPStore cases stay on cat 12 and the labeled progress-log assertions stay on
cat 21.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>